<html>
<head><meta charset="utf-8"><title>stats-for-differential-dataflow · t-compiler/wg-nll · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/index.html">t-compiler/wg-nll</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html">stats-for-differential-dataflow</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="126538482"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126538482" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126538482">(May 14 2018 at 13:50)</a>:</h4>
<p>So the discussion over in the <a href="#narrow/stream/122657-wg-nll/subject/optimizing-transitive-closure" title="#narrow/stream/122657-wg-nll/subject/optimizing-transitive-closure">optimizing-transitive-closure thread</a> got me thinking. Maybe we should alter the <code>-z</code> stats option to gather up internal statistics about the differential-dataflow computation itself (<span class="user-mention" data-user-id="116013">@qmx</span> this might be nice "in road" for playing around with it; <span class="user-mention" data-user-id="116113">@lqd</span> you might enjoy too). The idea would be fairly simple: we would insert the <code>count</code> operations at strategic points and dump out the results, trying to get a feel for how many tuples are flowing around at each point. I'm not sure that the <code>hist</code> package is very useful for this, though, we'd probably want to dump the results more in tabular form.</p>



<a name="126538688"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126538688" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126538688">(May 14 2018 at 13:55)</a>:</h4>
<p>possibly related to <a href="https://github.com/rust-lang-nursery/borrow-check/issues/12" target="_blank" title="https://github.com/rust-lang-nursery/borrow-check/issues/12">https://github.com/rust-lang-nursery/borrow-check/issues/12</a> but just with total numbers (as dumping 18M triples in a graph would be hard to read :)</p>



<a name="126538742"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126538742" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126538742">(May 14 2018 at 13:56)</a>:</h4>
<p>indeed</p>



<a name="126538746"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126538746" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126538746">(May 14 2018 at 13:56)</a>:</h4>
<p>I should probably merge my branch</p>



<a name="126538749"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126538749" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126538749">(May 14 2018 at 13:56)</a>:</h4>
<p>since obviously it is the way to go</p>



<a name="126538751"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126538751" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126538751">(May 14 2018 at 13:56)</a>:</h4>
<p><a href="https://github.com/rust-lang-nursery/borrow-check/pull/23" target="_blank" title="https://github.com/rust-lang-nursery/borrow-check/pull/23">https://github.com/rust-lang-nursery/borrow-check/pull/23</a> I mean</p>



<a name="126538753"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126538753" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126538753">(May 14 2018 at 13:56)</a>:</h4>
<p>though I ought to fix that assertion</p>



<a name="126538770"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126538770" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126538770">(May 14 2018 at 13:57)</a>:</h4>
<p>agreed, it improves things so much <span class="emoji emoji-1f44d" title="thumbs up">:thumbs_up:</span></p>



<a name="126539140"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126539140" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126539140">(May 14 2018 at 14:05)</a>:</h4>
<p>to be honest, I think an immediate next step would be cleaning up the code there</p>



<a name="126539143"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126539143" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126539143">(May 14 2018 at 14:05)</a>:</h4>
<p>there are a number of repetitive patterns that I was too lazy to abstract</p>



<a name="126539146"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126539146" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126539146">(May 14 2018 at 14:05)</a>:</h4>
<p>for example:</p>



<a name="126539150"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126539150" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126539150">(May 14 2018 at 14:05)</a>:</h4>
<div class="codehilite"><pre><span></span><span class="w">                        </span><span class="kd">let</span><span class="w"> </span><span class="n">dead_can_reach</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Variable</span>::<span class="n">from</span><span class="p">(</span><span class="n">Collection</span>::<span class="n">new</span><span class="p">(</span><span class="w"></span>
<span class="w">                            </span><span class="nb">None</span>::<span class="o">&lt;</span><span class="p">(((</span><span class="n">Region</span><span class="p">,</span><span class="w"> </span><span class="n">Point</span><span class="p">),</span><span class="w"> </span><span class="p">(</span><span class="n">Region</span><span class="p">,</span><span class="w"> </span><span class="n">Point</span><span class="p">)),</span><span class="w"> </span><span class="n">_</span><span class="p">,</span><span class="w"> </span><span class="n">_</span><span class="p">)</span><span class="o">&gt;</span><span class="w"></span>
<span class="w">                                </span><span class="p">.</span><span class="n">into_iter</span><span class="p">()</span><span class="w"></span>
<span class="w">                                </span><span class="p">.</span><span class="n">to_stream</span><span class="p">(</span><span class="n">subscope</span><span class="p">),</span><span class="w"></span>
<span class="w">                        </span><span class="p">));</span><span class="w"></span>
</pre></div>



<a name="126539155"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126539155" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126539155">(May 14 2018 at 14:06)</a>:</h4>
<p>it would be nice to make some sort of <code>let dead_can_reach = Variable::empty();</code> instead</p>



<a name="126539205"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126539205" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126539205">(May 14 2018 at 14:06)</a>:</h4>
<p>or maybe:</p>
<div class="codehilite"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="n">dead_can_reach</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Variable</span>:<span class="o">&lt;</span><span class="p">((</span><span class="n">Region</span><span class="p">,</span><span class="w"> </span><span class="n">Point</span><span class="p">),</span><span class="w"> </span><span class="p">(</span><span class="n">Region</span><span class="p">,</span><span class="w"> </span><span class="n">Point</span><span class="p">))</span><span class="o">&gt;</span>::<span class="n">empty</span><span class="p">();</span><span class="w"></span>
</pre></div>


<p>(I actually <em>like</em> declaring the types here)</p>



<a name="126539228"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126539228" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126539228">(May 14 2018 at 14:07)</a>:</h4>
<p>similarly all the calls to <code>join_core</code> would be nicer if abstracted</p>



<a name="126541073"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126541073" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126541073">(May 14 2018 at 14:51)</a>:</h4>
<p>Should be mostly easy to put in a <code>Variable::empty()</code> or <code>Variable::new()</code>, but it will need some handle to the scope. It is currently getting this from the empty input collection it is handed, but will otherwise need a handle in order to create the correct internal timely dataflow structures.</p>



<a name="126541279"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126541279" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126541279">(May 14 2018 at 14:54)</a>:</h4>
<p>yeah, I was just sketching...</p>



<a name="126541284"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126541284" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126541284">(May 14 2018 at 14:54)</a>:</h4>
<p>I need to open an issue for that stuff</p>



<a name="126546402"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546402" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546402">(May 14 2018 at 16:44)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> did you want counts for only the main relations (region_live_at/subset/requires) or for the subrelations as well (which would require aggregation over the timely rounds I think) ?</p>



<a name="126546413"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546413" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546413">(May 14 2018 at 16:45)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> seems like we could start with just the "main" relations</p>



<a name="126546417"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546417" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546417">(May 14 2018 at 16:45)</a>:</h4>
<p>that said, I'd be sort of interested in all of them I imagine</p>



<a name="126546421"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546421" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546421">(May 14 2018 at 16:45)</a>:</h4>
<p>I think it.. would probably require aggregation? I'm not sure I understand that part yet</p>



<a name="126546423"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546423" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546423">(May 14 2018 at 16:45)</a>:</h4>
<p>specifically, I don't know if a call to <code>consolidate()</code> would suffice</p>



<a name="126546472"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546472" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546472">(May 14 2018 at 16:46)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> can probably answer — specifically, if we call <code>consolidate()</code>, but there are internal rounds (due to a variable, etc), will we still have to sum over those rounds? That said, it seems...kind of interesting to see the <em>full</em> counts</p>



<a name="126546473"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546473" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546473">(May 14 2018 at 16:46)</a>:</h4>
<p>i.e., what we want is kind of a measure of how much "work" is happening?</p>



<a name="126546487"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546487" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546487">(May 14 2018 at 16:47)</a>:</h4>
<p>from the limited testing I did, if I put the consolidate + inspect inside the "subscope" I'd see more than one count output</p>



<a name="126546570"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546570" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546570">(May 14 2018 at 16:49)</a>:</h4>
<p>if you want the main relations totals before the PR, here it is :p <br>
'region_live_at' tuple counts: 1076158<br>
'subset' tuple counts: 2714121<br>
'requires' tuple counts: 858674</p>



<a name="126546659"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546659" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546659">(May 14 2018 at 16:51)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> thanks; <code>region_live_at</code> is the "input" tuples anyway, right? (but still interesting)</p>



<a name="126546666"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546666" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546666">(May 14 2018 at 16:51)</a>:</h4>
<p>it'd be good to also print out how many <code>Point</code> and <code>Region</code> instances there are</p>



<a name="126546670"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546670" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546670">(May 14 2018 at 16:51)</a>:</h4>
<p>oh, bother, I had some changes to the <code>Intern</code> to make that easy</p>



<a name="126546711"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546711" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546711">(May 14 2018 at 16:52)</a>:</h4>
<p>oh yes true, it's just in the verbose output for some reason, so I did those 3</p>



<a name="126546723"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546723" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546723">(May 14 2018 at 16:52)</a>:</h4>
<p>I also wanted (and did before) add a number of points + borrow_live_at sum for the --skip-tuples case</p>



<a name="126546799"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126546799" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126546799">(May 14 2018 at 16:54)</a>:</h4>
<p>eg: Output: 48992 keys, 832392 values</p>



<a name="126547369"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126547369" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126547369">(May 14 2018 at 17:07)</a>:</h4>
<p>Re: <code>consolidate()</code>, you can put that measurement fragment in a few places: </p>
<p>1. outside the loop that defines each of the relations (if you <code>.leave()</code> either of them), <br>
2. inside the loop after the <code>distinct_total()</code> (which will report new facts each round; if you print out the <code>.1</code> field in addition to <code>.2</code> it will report the "timestamp" == round of derivation), <br>
3. inside the loop before the <code>distinct_total()</code> which should show you the work going <em>in</em> to the distinct, which might reveal large numbers of redundant derivations that the distinct suppresses (or confirm that they do not exist).</p>



<a name="126547466"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126547466" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126547466">(May 14 2018 at 17:09)</a>:</h4>
<p>What <code>consolidate</code> does is flatten the collection down to at most one <code>(data, time)</code> pair for each distinct <code>data</code> and <code>time</code>, consolidating up the <code>diff</code> fields. When you map the <code>data</code> to <code>()</code> you just get the distinct <code>time</code>s which are rounds of derivation, and the accumulated <code>diff</code> which should be their count (either pre- or post-distinct).</p>



<a name="126548416"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126548416" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126548416">(May 14 2018 at 17:31)</a>:</h4>
<p>Another thing I can try for DD is to sort out the timely logging infra. In principle when enabled it produces all of the record counts and time spent in each operator across the computation. There is some work to do to synthesize it back into something intelligible (it will be a lot of "Join" and "Map"), but it seems like something I should do, and would probably be a lot more helpful than having you all place random measurement code everywhere.</p>
<p>I'll look into this and see what I can get quickly.</p>



<a name="126549523"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126549523" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126549523">(May 14 2018 at 17:55)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> for the timestamp, we should we print the <code>inner</code> timestamp right ? (rn it's <code>((Root, ()), 1)</code> but I assume it's because of the subscopes)</p>



<a name="126549527"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126549527" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126549527">(May 14 2018 at 17:55)</a>:</h4>
<p>Ok, I've got vanilla timely up and producing logs about what's been run and for how long. It shouldn't be too hard to aggregate times by operator and message count by edge. I'm going to look into whether DD can help put names on all of the operators and such, to try and clean up the presentation.</p>



<a name="126549587"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126549587" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126549587">(May 14 2018 at 17:56)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> yeah, the <code>.inner</code> field is the one you want. It should be the iteration counter, where the outer <code>()</code> is the empty timestamp indicating that no you cannot change the inputs at all (gives access to <code>distinct_total</code>).</p>



<a name="126549606"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126549606" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126549606">(May 14 2018 at 17:56)</a>:</h4>
<p>awesome, thank you :)</p>



<a name="126550081"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126550081" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126550081">(May 14 2018 at 18:08)</a>:</h4>
<p>I'm going to grab some food, but I wanted to drop some notes on the logging stuff. It may ultimately be as helpful as custom instrumentation (more flexible, for sure). I put a PR up at </p>
<p><a href="https://github.com/frankmcsherry/timely-dataflow/pull/159" target="_blank" title="https://github.com/frankmcsherry/timely-dataflow/pull/159">https://github.com/frankmcsherry/timely-dataflow/pull/159</a></p>
<p>which outlines some of the things, but the main gist is that you get a live timely dataflow stream of all the logging events, and can then whip up your own computation that consumes them (e.g. a DD computation that maintains accumulated counts and joins against the constructed names to produce more appealing reports). </p>
<p>I'll try and put together a prototype that may do a fine job of reporting record counts on every edge in the dataflow graph, without further modifying the computation (and without working around the measurement code). The downside is that it only sees record counts, and doesn't get to peek inside. If you have one record with 10M derivations, meaning a <code>.2</code> field of 10M, that will be one record possibly.</p>



<a name="126550764"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126550764" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126550764">(May 14 2018 at 18:22)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> gtg for a while, but I have something like this atm <a href="https://gist.github.com/lqd/f64f2c5cd338892044ecf145a3dd51c8" target="_blank" title="https://gist.github.com/lqd/f64f2c5cd338892044ecf145a3dd51c8">https://gist.github.com/lqd/f64f2c5cd338892044ecf145a3dd51c8</a> -- the first file is <code>--stats -v</code> to have the timely rounds stats, and the second one just <code>--stats</code></p>



<a name="126557032"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126557032" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126557032">(May 14 2018 at 20:39)</a>:</h4>
<p>Good news and bad news!</p>
<p>Good news is that I have a differential dataflow program that consumes a live timely logging stream and produces counts for records on all edges, described by the names of the operators.</p>
<p>Bad news is that it is lots of maps and joins and stuff. Here it is on the current BorrowCheck execution:</p>
<p><a href="https://gist.github.com/frankmcsherry/d72049d8e8ce88f89622cf48335f5e65" target="_blank" title="https://gist.github.com/frankmcsherry/d72049d8e8ce88f89622cf48335f5e65">https://gist.github.com/frankmcsherry/d72049d8e8ce88f89622cf48335f5e65</a></p>



<a name="126557133"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126557133" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126557133">(May 14 2018 at 20:42)</a>:</h4>
<p>I'm not sure I've really got the name &lt;--&gt; channel mapping nailed down perfectly (more research to do here), but you can probably start to get the gist. The only change needed in the borrow_check program is to initiate the timely computation like so:</p>
<div class="codehilite"><pre><span></span><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">logger_config</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">timely</span>::<span class="n">logging</span>::<span class="n">LoggerConfig</span>::<span class="n">new</span><span class="p">(</span><span class="w"></span>
<span class="w">        </span><span class="o">|</span><span class="n">setup</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="n">addr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">format</span><span class="o">!</span><span class="p">(</span><span class="s">&quot;127.0.0.1:{}&quot;</span><span class="p">,</span><span class="w"> </span><span class="mi">8000</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">setup</span><span class="p">.</span><span class="n">index</span><span class="p">);</span><span class="w"></span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="n">send</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>::<span class="n">std</span>::<span class="n">net</span>::<span class="n">TcpStream</span>::<span class="n">connect</span><span class="p">(</span><span class="n">addr</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w">            </span><span class="n">timely</span>::<span class="n">dataflow</span>::<span class="n">operators</span>::<span class="n">capture</span>::<span class="n">EventWriter</span>::<span class="n">new</span><span class="p">(</span><span class="n">send</span><span class="p">)</span><span class="w"></span>
<span class="w">        </span><span class="p">},</span><span class="w"></span>
<span class="w">        </span><span class="o">|</span><span class="n">_setup</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">            </span><span class="c1">// No support for communication threads in this example.</span>
<span class="w">            </span><span class="n">unimplemented</span><span class="o">!</span><span class="p">();</span><span class="w"></span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="n">addr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">format</span><span class="o">!</span><span class="p">(</span><span class="s">&quot;127.0.0.1:{}&quot;</span><span class="p">,</span><span class="w"> </span><span class="mi">8001</span><span class="p">);</span><span class="w"></span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="n">send</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>::<span class="n">std</span>::<span class="n">net</span>::<span class="n">TcpStream</span>::<span class="n">connect</span><span class="p">(</span><span class="n">addr</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w">            </span><span class="n">timely</span>::<span class="n">dataflow</span>::<span class="n">operators</span>::<span class="n">capture</span>::<span class="n">EventWriter</span>::<span class="n">new</span><span class="p">(</span><span class="n">send</span><span class="p">)</span><span class="w"></span>
<span class="w">        </span><span class="p">}</span><span class="w"></span>
<span class="w">    </span><span class="p">);</span><span class="w"></span>

<span class="w">    </span><span class="n">timely</span>::<span class="n">execute_from_args_logging</span><span class="p">(</span><span class="n">vec</span><span class="o">!</span><span class="p">[].</span><span class="n">into_iter</span><span class="p">(),</span><span class="w"> </span><span class="n">logger_config</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">result</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">result</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
</pre></div>


<p>which yes uses a TCP socket to grab the logging data. In principle it can go to any <code>W: Write</code>, or Kafka, or a bunch of other places.</p>



<a name="126557213"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126557213" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126557213">(May 14 2018 at 20:44)</a>:</h4>
<p>Another downside, unfortunately: the arrangement infrastructure is hiding some of the sizes. The records coming <em>out</em> of arrangements are the number of batches, rather than the number of records in the batch. This means that getting the details out of records into a distinct vs records out of a distinct means we have to look at the input to the arrange leading into the distinct, rather than just the distinct itself.</p>



<a name="126558699"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126558699" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126558699">(May 14 2018 at 21:16)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> here's the commit for the stats I posted earlier if you want a PR (10 min compile times on this tiny laptop are bru·tal :) — and then we can integrate frank's new shinies later ? <a href="https://github.com/lqd/borrow-check/commit/bb47d049ff04aebff079df2181ff2df5db909d31" target="_blank" title="https://github.com/lqd/borrow-check/commit/bb47d049ff04aebff079df2181ff2df5db909d31">https://github.com/lqd/borrow-check/commit/bb47d049ff04aebff079df2181ff2df5db909d31</a></p>



<a name="126558731"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126558731" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126558731">(May 14 2018 at 21:17)</a>:</h4>
<p>btw, the change proposed up above should be conditional somehow. It will panic if there isn't anyone listening on that port, and you probably don't want that most of the time (also, some run-time overhead for teh logging).</p>



<a name="126559218"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126559218" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126559218">(May 14 2018 at 21:28)</a>:</h4>
<p>It's midnight here, but if you want to check it out, </p>
<p><a href="https://github.com/frankmcsherry/differential-dataflow/blob/master/examples/logging-recv.rs" target="_blank" title="https://github.com/frankmcsherry/differential-dataflow/blob/master/examples/logging-recv.rs">https://github.com/frankmcsherry/differential-dataflow/blob/master/examples/logging-recv.rs</a></p>
<p>is the code for the log analysis. It's all just joining together things, and a bunch of consulting <a href="http://logging.rs" target="_blank" title="http://logging.rs">logging.rs</a> to remember which fields are which. More work to do to make it intelligible, and perhaps some thinking about how to better allow naming / grouping of operators in code so that the stats can be better interpreted.</p>



<a name="126559585"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126559585" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126559585">(May 14 2018 at 21:36)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> very nice :)</p>



<a name="126596641"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126596641" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126596641">(May 15 2018 at 15:18)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> btw I've not had a chance to look over what you did yet — will try to do that now</p>



<a name="126598018"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow/near/126598018" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/stats-for-differential-dataflow.html#126598018">(May 15 2018 at 15:46)</a>:</h4>
<p>no worries :)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>